Compression Details
Detailed token visualization per fixture.
Comparisons
Select tokenizer and example to compare formats
Compare minemizer to other encoding formats for LLM token efficiency.
Normalized comparison across formats and tokenizers (JSON pretty = 1.0x).
Normalized (JSON pretty = 1.0x, higher is better)
| Format | flat | nested | lists | sparse | complex | books | countries | large_mixed | large_num | large_text | mcp_tools | avg |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| JSON (pretty) | 1.0x | 1.0x | 1.0x | 1.0x | 1.0x | 1.0x | 1.0x | 1.0x | 1.0x | 1.0x | 1.0x | 1.0x |
| JSON (min) | 2.1x | 2.2x | 2.4x | 2.0x | 2.2x | 1.5x | 1.5x | 2.1x | 1.7x | 1.7x | 2.2x | 2.0x |
| CSV | 3.4x | ✗ | ✗ | ✗ | ✗ | 2.0x | ✗ | ✗ | ✗ | ✗ | ✗ | 2.7x |
| TSV | 3.4x | ✗ | ✗ | ✗ | ✗ | 1.9x | ✗ | ✗ | ✗ | ✗ | ✗ | 2.7x |
| YAML | 1.8x | 1.8x | 1.8x | 1.8x | 1.7x | 1.3x | 2.2x | 1.6x | 1.4x | 1.5x | 1.5x | 1.7x |
| TOON | 3.2x | 1.7x | 1.9x | 1.6x | 1.6x | 2.0x | 2.1x | 1.5x | 1.3x | 1.5x | 1.5x | 1.8x |
| TSON | 3.6x | 3.4x | 3.7x | 2.0x | 2.6x | 2.0x | 3.0x | 1.9x | 1.6x | 1.6x | 2.3x | 2.5x |
| minemizer | 4.1x | 3.4x | 3.7x | 3.6x | 3.0x | 2.0x | 3.9x | 2.4x | 1.8x | 2.2x | 2.8x | 3.0x |
| minemizer (compact) | 3.7x | 3.4x | 3.5x | 3.4x | 3.0x | 2.0x | 3.7x | 2.4x | 1.9x | 2.1x | 2.8x | 2.9x |
Average efficiency per tokenizer (normalized)
| Format | gpt2 | llama | qwen2.5 | deepseek | devstral | avg |
|---|---|---|---|---|---|---|
| JSON (pretty) | 1.0x | 1.0x | 1.0x | 1.0x | 1.0x | 1.0x |
| JSON (min) | 2.7x | 1.8x | 1.7x | 1.7x | 1.9x | 2.0x |
| CSV | 3.2x | 2.6x | 2.7x | 2.5x | 2.7x | 2.7x |
| TSV | 3.1x | 2.6x | 2.7x | 2.4x | 2.7x | 2.7x |
| YAML | 2.0x | 1.7x | 1.5x | 1.5x | 1.7x | 1.7x |
| TOON | 2.0x | 1.9x | 1.7x | 1.7x | 1.9x | 1.8x |
| TSON | 3.2x | 2.4x | 2.3x | 2.2x | 2.5x | 2.5x |
| minemizer | 4.0x | 2.8x | 2.6x | 2.6x | 3.0x | 3.0x |
| minemizer (compact) | 3.8x | 2.7x | 2.6x | 2.6x | 2.9x | 2.9x |
7 dataset(s), 6 model(s). Sorted by efficiency.
| Format | Efficiency | Accuracy | Tokens | og_chars/tok | Latency |
|---|---|---|---|---|---|
| minemizer_compact | 1.36 | 53.5% | 24.1k | 7.0 | 3271ms |
| minemizer | 1.34 | 54.7% | 25.1k | 6.7 | 3349ms |
| minemizer_compact_no_repeat | 1.33 | 51.8% | 24.0k | 7.1 | 3106ms |
| minemizer_no_repeat | 1.27 | 51.7% | 25.0k | 6.8 | 3193ms |
| tson | 0.83 | 30.5% | 22.6k | 7.5 | 3025ms |
| yaml | 0.81 | 57.4% | 43.6k | 3.9 | 6662ms |
| json_min | 0.74 | 44.5% | 36.8k | 4.6 | 4581ms |
| toon | 0.73 | 52.9% | 44.3k | 3.8 | 6647ms |
| json_pretty | 0.52 | 52.2% | 61.5k | 2.8 | 4620ms |
2 dataset(s), 6 model(s). Sorted by efficiency.
| Format | Efficiency | Accuracy | Tokens | og_chars/tok | Latency |
|---|---|---|---|---|---|
| csv | 1.66 | 72.5% | 6.5k | 5.7 | 787ms |
| minemizer_compact | 1.61 | 70.5% | 6.6k | 5.7 | 739ms |
| toon | 1.61 | 71.0% | 6.6k | 5.6 | 787ms |
| tsv | 1.60 | 70.5% | 6.6k | 5.7 | 779ms |
| minemizer_compact_no_repeat | 1.58 | 69.0% | 6.5k | 5.7 | 737ms |
| minemizer | 1.47 | 66.5% | 6.8k | 5.5 | 732ms |
| minemizer_no_repeat | 1.44 | 65.0% | 6.8k | 5.5 | 743ms |
| tson | 1.35 | 59.0% | 6.5k | 5.7 | 787ms |
| yaml | 1.07 | 82.0% | 11.5k | 3.2 | 807ms |
| minemizer_compact_repeat10 | 0.87 | 57.0% | 9.8k | 3.8 | 848ms |
| minemizer_repeat10 | 0.86 | 58.0% | 10.1k | 3.7 | 851ms |
| json_min | 0.75 | 51.0% | 10.2k | 3.6 | 956ms |
| json_pretty | 0.60 | 60.5% | 15.0k | 2.5 | 692ms |
1 dataset(s), 6 model(s). Sorted by efficiency.
| Format | Efficiency | Accuracy | Tokens | og_chars/tok | Latency |
|---|---|---|---|---|---|
| minemizer_no_repeat | 1.26 | 61.0% | 16.5k | 5.6 | 905ms |
| minemizer_repeat10 | 1.24 | 62.0% | 17.0k | 5.4 | 965ms |
| minemizer_compact_repeat10 | 1.19 | 60.0% | 17.2k | 5.4 | 954ms |
| minemizer | 1.15 | 56.0% | 16.6k | 5.6 | 919ms |
| minemizer_compact_no_repeat | 1.15 | 56.0% | 16.6k | 5.6 | 934ms |
| minemizer_compact | 1.11 | 54.0% | 16.6k | 5.6 | 905ms |
| tson | 0.84 | 55.0% | 22.3k | 4.2 | 1009ms |
| yaml | 0.83 | 60.0% | 24.8k | 3.7 | 1075ms |
| json_min | 0.70 | 46.0% | 22.3k | 4.2 | 1131ms |
| toon | 0.69 | 49.0% | 24.3k | 3.8 | 1048ms |
| json_pretty | 0.42 | 42.0% | 34.1k | 2.7 | 1140ms |
Efficiency = Accuracy × (JSON tokens ÷ Format tokens). Higher is better — balances accuracy with token savings.
Detailed token visualization per fixture.
Select tokenizer and example to compare formats
Format comprehension across models and datasets.
| Format | Efficiency | Accuracy | Tokens | og_chars/tok | Latency |
|---|---|---|---|---|---|
| minemizer_compact | 1.97 | 81.8% | 2,701 | 6.4 | 684ms |
| minemizer_compact_no_repeat | 1.97 | 81.8% | 2,701 | 6.4 | 683ms |
| minemizer | 1.89 | 81.0% | 2,796 | 6.2 | 653ms |
| minemizer_no_repeat | 1.89 | 81.0% | 2,796 | 6.2 | 650ms |
| yaml | 1.18 | 84.7% | 4,673 | 3.7 | 755ms |
| toon | 1.08 | 78.8% | 4,752 | 3.6 | 796ms |
| json_min | 1.00 | 63.3% | 4,109 | 4.2 | 818ms |
| tson | 0.85 | 34.8% | 2,655 | 6.5 | 708ms |
| json_pretty | 0.80 | 79.8% | 6,510 | 2.7 | 816ms |
| Format | Efficiency | Accuracy | Tokens | og_chars/tok | Latency |
|---|---|---|---|---|---|
| csv | 1.66 | 72.5% | 6,545 | 5.7 | 787ms |
| minemizer_compact | 1.61 | 70.5% | 6,556 | 5.7 | 739ms |
| toon | 1.61 | 71.0% | 6,614 | 5.6 | 787ms |
| tsv | 1.60 | 70.5% | 6,582 | 5.7 | 779ms |
| minemizer_compact_no_repeat | 1.58 | 69.0% | 6,536 | 5.7 | 737ms |
| minemizer | 1.47 | 66.5% | 6,776 | 5.5 | 732ms |
| minemizer_no_repeat | 1.44 | 65.0% | 6,757 | 5.5 | 743ms |
| tson | 1.35 | 59.0% | 6,547 | 5.7 | 787ms |
| yaml | 1.07 | 82.0% | 11.5k | 3.2 | 807ms |
| minemizer_compact_repeat10 | 0.87 | 57.0% | 9,783 | 3.8 | 848ms |
| minemizer_repeat10 | 0.86 | 58.0% | 10.1k | 3.7 | 851ms |
| json_min | 0.75 | 51.0% | 10.2k | 3.6 | 956ms |
| json_pretty | 0.60 | 60.5% | 15.0k | 2.5 | 692ms |
| Format | Efficiency | Accuracy | Tokens | og_chars/tok | Latency |
|---|---|---|---|---|---|
| minemizer_no_repeat | 1.26 | 61.0% | 16.5k | 5.6 | 905ms |
| minemizer_repeat10 | 1.24 | 62.0% | 17.0k | 5.4 | 965ms |
| minemizer_compact_repeat10 | 1.19 | 60.0% | 17.2k | 5.4 | 954ms |
| minemizer | 1.15 | 56.0% | 16.6k | 5.6 | 919ms |
| minemizer_compact_no_repeat | 1.15 | 56.0% | 16.6k | 5.6 | 934ms |
| minemizer_compact | 1.11 | 54.0% | 16.6k | 5.6 | 905ms |
| tson | 0.84 | 55.0% | 22.3k | 4.2 | 1009ms |
| yaml | 0.83 | 60.0% | 24.8k | 3.7 | 1075ms |
| json_min | 0.70 | 46.0% | 22.3k | 4.2 | 1131ms |
| toon | 0.69 | 49.0% | 24.3k | 3.8 | 1048ms |
| json_pretty | 0.42 | 42.0% | 34.1k | 2.7 | 1140ms |
| Dataset | Records | Queries | Date | Best Format | Best Acc |
|---|---|---|---|---|---|
| flat_100 | 100 | 100 | 2025-12-13 | yaml | 90.0% |
| flat_250 | 250 | 100 | 2025-12-13 | yaml | 74.0% |
| nested_100 | 100 | 100 | 2025-12-12 | yaml | 82.0% |
| nested_50 | 50 | 200 | 2025-12-12 | json_pretty | 90.5% |
| sparse_250 | 250 | 100 | 2025-12-13 | minemizer_repeat10 | 62.0% |
| Format | Efficiency | Accuracy | Tokens | og_chars/tok | Latency |
|---|---|---|---|---|---|
| minemizer_compact_no_repeat | 1.05 | 38.0% | 28.9k | 8.0 | 1428ms |
| minemizer_compact | 1.05 | 38.0% | 29.0k | 7.9 | 1615ms |
| minemizer | 0.94 | 36.0% | 30.5k | 7.5 | 1998ms |
| tson | 0.94 | 32.0% | 27.3k | 8.4 | 1708ms |
| minemizer_no_repeat | 0.79 | 30.0% | 30.4k | 7.6 | 1687ms |
| json_pretty | 0.36 | 36.0% | 79.8k | 2.9 | 6448ms |
| toon | 0.25 | 18.0% | 56.6k | 4.1 | 4035ms |
| yaml | 0.17 | 12.0% | 55.6k | 4.1 | 4110ms |
| json_min | 0.07 | 4.0% | 45.8k | 5.0 | 3493ms |
| Dataset | Records | Queries | Date | Best Format | Best Acc |
|---|---|---|---|---|---|
| nested_1000 | 1,000 | 50 | 2025-12-02 | minemizer_compact | 38.0% |
| Format | Efficiency | Accuracy | Tokens | og_chars/tok | Latency |
|---|---|---|---|---|---|
| minemizer_compact | 1.59 | 65.0% | 35.1k | 6.6 | 8172ms |
| minemizer | 1.54 | 65.0% | 36.3k | 6.3 | 8367ms |
| minemizer_compact_no_repeat | 1.35 | 55.0% | 34.9k | 6.6 | 7569ms |
| minemizer_no_repeat | 1.19 | 50.0% | 36.2k | 6.4 | 7958ms |
| yaml | 1.12 | 80.0% | 61.4k | 3.7 | 18577ms |
| json_min | 1.08 | 65.0% | 51.9k | 4.4 | 11098ms |
| toon | 0.89 | 65.0% | 62.4k | 3.7 | 19448ms |
| tson | 0.80 | 30.0% | 32.1k | 7.2 | 7133ms |
| json_pretty | 0.50 | 50.0% | 85.9k | 2.7 | 2795ms |
| Dataset | Records | Queries | Date | Best Format | Best Acc |
|---|---|---|---|---|---|
| nested_1000 | 1,000 | 20 | 2025-12-01 | yaml | 80.0% |
| Format | Efficiency | Accuracy | Tokens | og_chars/tok | Latency |
|---|---|---|---|---|---|
| minemizer_compact | 1.37 | 56.0% | 35.1k | 6.6 | 3202ms |
| minemizer | 1.37 | 58.0% | 36.3k | 6.3 | 3175ms |
| tson | 1.34 | 50.0% | 32.1k | 7.2 | 2883ms |
| minemizer_no_repeat | 1.28 | 54.0% | 36.2k | 6.4 | 3166ms |
| minemizer_compact_no_repeat | 1.23 | 50.0% | 34.9k | 6.6 | 3080ms |
| yaml | 0.78 | 56.0% | 61.4k | 3.7 | 6600ms |
| json_min | 0.73 | 44.0% | 51.9k | 4.4 | 4151ms |
| toon | 0.63 | 46.0% | 62.4k | 3.7 | 6255ms |
| json_pretty | 0.40 | 40.0% | 85.9k | 2.7 | 10601ms |
| Dataset | Records | Queries | Date | Best Format | Best Acc |
|---|---|---|---|---|---|
| nested_1000 | 1,000 | 50 | 2025-12-02 | minemizer | 58.0% |
| Format | Efficiency | Accuracy | Tokens | og_chars/tok | Latency |
|---|---|---|---|---|---|
| minemizer | 0.84 | 32.0% | 30.6k | 7.5 | 2818ms |
| minemizer_no_repeat | 0.79 | 30.0% | 30.4k | 7.6 | 2691ms |
| json_min | 0.77 | 44.0% | 45.9k | 5.0 | 3073ms |
| toon | 0.70 | 50.0% | 56.7k | 4.1 | 3788ms |
| yaml | 0.69 | 48.0% | 55.7k | 4.1 | 3689ms |
| minemizer_compact_no_repeat | 0.55 | 20.0% | 28.9k | 8.0 | 2672ms |
| minemizer_compact | 0.49 | 18.0% | 29.1k | 7.9 | 2918ms |
| json_pretty | 0.44 | 44.0% | 79.9k | 2.9 | 5081ms |
| tson | 0.23 | 8.0% | 27.4k | 8.4 | 2804ms |
| Dataset | Records | Queries | Date | Best Format | Best Acc |
|---|---|---|---|---|---|
| nested_1000 | 1,000 | 50 | 2025-12-02 | toon | 50.0% |
| Format | Efficiency | Accuracy | Tokens | og_chars/tok | Latency |
|---|---|---|---|---|---|
| minemizer_compact_no_repeat | 0.88 | 36.0% | 35.1k | 6.6 | 5630ms |
| minemizer_no_repeat | 0.85 | 36.0% | 36.3k | 6.3 | 5552ms |
| minemizer_compact | 0.83 | 34.0% | 35.2k | 6.5 | 5626ms |
| minemizer | 0.71 | 30.0% | 36.4k | 6.3 | 5776ms |
| tson | 0.60 | 24.0% | 34.3k | 6.7 | 5230ms |
| yaml | 0.50 | 36.0% | 61.6k | 3.7 | 12152ms |
| toon | 0.47 | 34.0% | 62.6k | 3.7 | 11410ms |
| json_min | 0.45 | 28.0% | 54.0k | 4.3 | 8619ms |
| json_pretty | 0.36 | 36.0% | 86.0k | 2.7 | 5778ms |
| Dataset | Records | Queries | Date | Best Format | Best Acc |
|---|---|---|---|---|---|
| nested_1000 | 1,000 | 50 | 2025-12-12 | json_pretty | 36.0% |
| Format | Efficiency | Accuracy | Tokens | og_chars/tok | Latency |
|---|---|---|---|---|---|
| csv | 1.86 | 82.0% | 3,777 | 5.6 | 657ms |
| tsv | 1.83 | 81.0% | 3,798 | 5.6 | 647ms |
| minemizer_compact | 1.80 | 79.0% | 3,768 | 5.6 | 632ms |
| minemizer_compact_no_repeat | 1.80 | 79.0% | 3,768 | 5.6 | 619ms |
| toon | 1.80 | 80.0% | 3,821 | 5.6 | 660ms |
| minemizer | 1.61 | 73.0% | 3,897 | 5.5 | 622ms |
| minemizer_no_repeat | 1.61 | 73.0% | 3,897 | 5.5 | 630ms |
| tson | 1.57 | 69.0% | 3,777 | 5.6 | 658ms |
| yaml | 1.17 | 90.0% | 6,617 | 3.2 | 665ms |
| json_min | 0.89 | 61.0% | 5,880 | 3.6 | 838ms |
| json_pretty | 0.66 | 66.0% | 8,581 | 2.5 | 627ms |
Best variant: minemizer_compact
Token savings: 56.1%
Accuracy diff: +13.0%
| Format | Efficiency | Accuracy | Tokens | og_chars/tok | Latency |
|---|---|---|---|---|---|
| csv | 1.45 | 63.0% | 9,313 | 5.7 | 916ms |
| minemizer_compact | 1.42 | 62.0% | 9,343 | 5.7 | 846ms |
| toon | 1.41 | 62.0% | 9,406 | 5.7 | 915ms |
| tsv | 1.37 | 60.0% | 9,367 | 5.7 | 912ms |
| minemizer_compact_no_repeat | 1.36 | 59.0% | 9,303 | 5.7 | 855ms |
| minemizer | 1.33 | 60.0% | 9,655 | 5.5 | 841ms |
| minemizer_no_repeat | 1.27 | 57.0% | 9,617 | 5.5 | 855ms |
| minemizer_compact_repeat10 | 1.24 | 57.0% | 9,783 | 5.4 | 848ms |
| minemizer_repeat10 | 1.23 | 58.0% | 10,073 | 5.3 | 851ms |
| tson | 1.12 | 49.0% | 9,317 | 5.7 | 915ms |
| yaml | 0.96 | 74.0% | 16,436 | 3.2 | 948ms |
| json_min | 0.60 | 41.0% | 14,615 | 3.6 | 1075ms |
| json_pretty | 0.55 | 55.0% | 21,366 | 2.5 | 758ms |
Best variant: minemizer_compact
Token savings: 56.3%
Accuracy diff: +7.0%
| Format | Efficiency | Accuracy | Tokens | og_chars/tok | Latency |
|---|---|---|---|---|---|
| minemizer_compact | 1.84 | 76.0% | 3,568 | 6.5 | 755ms |
| minemizer_compact_no_repeat | 1.84 | 76.0% | 3,568 | 6.5 | 754ms |
| minemizer | 1.83 | 78.0% | 3,695 | 6.2 | 716ms |
| minemizer_no_repeat | 1.83 | 78.0% | 3,695 | 6.2 | 704ms |
| json_min | 1.27 | 80.0% | 5,451 | 4.2 | 837ms |
| yaml | 1.14 | 82.0% | 6,204 | 3.7 | 861ms |
| toon | 1.03 | 75.0% | 6,309 | 3.6 | 804ms |
| tson | 0.82 | 33.0% | 3,502 | 6.6 | 753ms |
| json_pretty | 0.69 | 69.0% | 8,652 | 2.7 | 878ms |
| csv | - | - | - | - | - |
| tsv | - | - | - | - | - |
Best variant: minemizer
Token savings: 57.3%
Accuracy diff: +9.0%
| Format | Efficiency | Accuracy | Tokens | og_chars/tok | Latency |
|---|---|---|---|---|---|
| minemizer_compact | 2.08 | 87.5% | 1,834 | 6.3 | 612ms |
| minemizer_compact_no_repeat | 2.08 | 87.5% | 1,834 | 6.3 | 612ms |
| minemizer | 1.94 | 84.0% | 1,896 | 6.1 | 591ms |
| minemizer_no_repeat | 1.94 | 84.0% | 1,896 | 6.1 | 596ms |
| yaml | 1.22 | 87.5% | 3,142 | 3.7 | 648ms |
| toon | 1.13 | 82.5% | 3,196 | 3.6 | 788ms |
| json_pretty | 0.91 | 90.5% | 4,368 | 2.6 | 754ms |
| tson | 0.88 | 36.5% | 1,808 | 6.4 | 663ms |
| json_min | 0.73 | 46.5% | 2,767 | 4.2 | 799ms |
| csv | - | - | - | - | - |
| tsv | - | - | - | - | - |
Best variant: minemizer_compact
Token savings: 58.0%
Accuracy diff: -3.0%
| Format | Efficiency | Accuracy | Tokens | og_chars/tok | Latency |
|---|---|---|---|---|---|
| minemizer_no_repeat | 1.26 | 61.0% | 16,517 | 5.6 | 905ms |
| minemizer_repeat10 | 1.24 | 62.0% | 17,045 | 5.4 | 965ms |
| minemizer_compact_repeat10 | 1.19 | 60.0% | 17,172 | 5.4 | 954ms |
| minemizer | 1.15 | 56.0% | 16,561 | 5.6 | 919ms |
| minemizer_compact_no_repeat | 1.15 | 56.0% | 16,596 | 5.6 | 934ms |
| minemizer_compact | 1.11 | 54.0% | 16,644 | 5.6 | 905ms |
| tson | 0.84 | 55.0% | 22,255 | 4.2 | 1009ms |
| yaml | 0.83 | 60.0% | 24,766 | 3.7 | 1075ms |
| json_min | 0.70 | 46.0% | 22,302 | 4.2 | 1131ms |
| toon | 0.69 | 49.0% | 24,306 | 3.8 | 1048ms |
| json_pretty | 0.42 | 42.0% | 34,070 | 2.7 | 1140ms |
| csv | - | - | - | - | - |
| tsv | - | - | - | - | - |
Best variant: minemizer_repeat10
Token savings: 50.0%
Accuracy diff: +20.0%
| Format | Efficiency | Accuracy | Tokens | og_chars/tok | Latency |
|---|---|---|---|---|---|
| minemizer_compact_no_repeat | 1.05 | 38.0% | 28,858 | 8.0 | 1428ms |
| minemizer_compact | 1.05 | 38.0% | 29,011 | 7.9 | 1615ms |
| minemizer | 0.94 | 36.0% | 30,536 | 7.5 | 1998ms |
| tson | 0.94 | 32.0% | 27,315 | 8.4 | 1708ms |
| minemizer_no_repeat | 0.79 | 30.0% | 30,374 | 7.6 | 1687ms |
| json_pretty | 0.36 | 36.0% | 79,843 | 2.9 | 6448ms |
| toon | 0.25 | 18.0% | 56,631 | 4.1 | 4035ms |
| yaml | 0.17 | 12.0% | 55,627 | 4.1 | 4110ms |
| json_min | 0.07 | 4.0% | 45,843 | 5.0 | 3493ms |
| csv | - | - | - | - | - |
| tsv | - | - | - | - | - |
Best variant: minemizer_compact
Token savings: 63.7%
Accuracy diff: +2.0%
| Format | Efficiency | Accuracy | Tokens | og_chars/tok | Latency |
|---|---|---|---|---|---|
| minemizer_compact | 1.59 | 65.0% | 35,068 | 6.6 | 8172ms |
| minemizer | 1.54 | 65.0% | 36,341 | 6.3 | 8367ms |
| minemizer_compact_no_repeat | 1.35 | 55.0% | 34,915 | 6.6 | 7569ms |
| minemizer_no_repeat | 1.19 | 50.0% | 36,179 | 6.4 | 7958ms |
| yaml | 1.12 | 80.0% | 61,436 | 3.7 | 18577ms |
| json_min | 1.08 | 65.0% | 51,900 | 4.4 | 11098ms |
| toon | 0.89 | 65.0% | 62,442 | 3.7 | 19448ms |
| tson | 0.80 | 30.0% | 32,121 | 7.2 | 7133ms |
| json_pretty | 0.50 | 50.0% | 85,900 | 2.7 | 2795ms |
| csv | - | - | - | - | - |
| tsv | - | - | - | - | - |
Best variant: minemizer
Token savings: 57.7%
Accuracy diff: +15.0%
| Format | Efficiency | Accuracy | Tokens | og_chars/tok | Latency |
|---|---|---|---|---|---|
| minemizer_compact | 1.37 | 56.0% | 35,064 | 6.6 | 3202ms |
| minemizer | 1.37 | 58.0% | 36,337 | 6.3 | 3175ms |
| tson | 1.34 | 50.0% | 32,117 | 7.2 | 2883ms |
| minemizer_no_repeat | 1.28 | 54.0% | 36,175 | 6.4 | 3166ms |
| minemizer_compact_no_repeat | 1.23 | 50.0% | 34,911 | 6.6 | 3080ms |
| yaml | 0.78 | 56.0% | 61,432 | 3.7 | 6600ms |
| json_min | 0.73 | 44.0% | 51,896 | 4.4 | 4151ms |
| toon | 0.63 | 46.0% | 62,438 | 3.7 | 6255ms |
| json_pretty | 0.40 | 40.0% | 85,896 | 2.7 | 10601ms |
| csv | - | - | - | - | - |
| tsv | - | - | - | - | - |
Best variant: minemizer
Token savings: 57.7%
Accuracy diff: +18.0%
| Format | Efficiency | Accuracy | Tokens | og_chars/tok | Latency |
|---|---|---|---|---|---|
| minemizer | 0.84 | 32.0% | 30,600 | 7.5 | 2818ms |
| minemizer_no_repeat | 0.79 | 30.0% | 30,438 | 7.6 | 2691ms |
| json_min | 0.77 | 44.0% | 45,907 | 5.0 | 3073ms |
| toon | 0.70 | 50.0% | 56,695 | 4.1 | 3788ms |
| yaml | 0.69 | 48.0% | 55,691 | 4.1 | 3689ms |
| minemizer_compact_no_repeat | 0.55 | 20.0% | 28,922 | 8.0 | 2672ms |
| minemizer_compact | 0.49 | 18.0% | 29,075 | 7.9 | 2918ms |
| json_pretty | 0.44 | 44.0% | 79,907 | 2.9 | 5081ms |
| tson | 0.23 | 8.0% | 27,379 | 8.4 | 2804ms |
| csv | - | - | - | - | - |
| tsv | - | - | - | - | - |
Best variant: minemizer
Token savings: 61.7%
Accuracy diff: -12.0%
| Format | Efficiency | Accuracy | Tokens | og_chars/tok | Latency |
|---|---|---|---|---|---|
| minemizer_compact_no_repeat | 0.88 | 36.0% | 35,051 | 6.6 | 5630ms |
| minemizer_no_repeat | 0.85 | 36.0% | 36,276 | 6.3 | 5552ms |
| minemizer_compact | 0.83 | 34.0% | 35,213 | 6.5 | 5626ms |
| minemizer | 0.71 | 30.0% | 36,438 | 6.3 | 5776ms |
| tson | 0.60 | 24.0% | 34,303 | 6.7 | 5230ms |
| yaml | 0.50 | 36.0% | 61,559 | 3.7 | 12152ms |
| toon | 0.47 | 34.0% | 62,565 | 3.7 | 11410ms |
| json_min | 0.45 | 28.0% | 54,034 | 4.3 | 8619ms |
| json_pretty | 0.36 | 36.0% | 86,035 | 2.7 | 5778ms |
| csv | - | - | - | - | - |
| tsv | - | - | - | - | - |
Best variant: minemizer_no_repeat
Token savings: 57.8%
Accuracy diff: +0.0%